Before you turn this problem in, make sure everything runs as expected. First, restart the kernel (in the menubar, select Kernel$\rightarrow$Restart) and then run all cells (in the menubar, select Cell$\rightarrow$Run All).

If a variable is given to you, give your answer to that variable. In case of multiple choice questions, answer to the given variable. Make sure your variable has a string value and it contains only the letter of your choice. For example if you think that choice b is the correct choice, and the given variable is answer, your answer should look like this:

answer = 'b'

or

answer = 'B'

If you need to read a file for a task, datafiles, such as .csv or .txt files are found in data subfolder. This can be accessed using data/filename.extension. In case of image, they are found in subfolder images.

Make sure you fill in any place that says YOUR CODE HERE or "YOUR ANSWER HERE", as well as your name below:

NAME = "Houssem Menhour"

Multimodal Data Fusion - Exercise 3 - Data Alignment¶

In this exercise, we will take a closer look at the alignment of (multimodal) data in spatial, temporal, and semantic domains as well as radiometric normalization of calibrating the scale of variables. You will be learning about spatial alignment with image registration, temporal alignment with dynamic time warping of time-series data, semantical aligment of different data sources with clustering methods for image segmentation, and data normalization having different scales.

Learning goals¶

After this exercise, you should

get familiar with different data alignment types and techniques in multimodal data fusion
be able to estimate of mutual information for spatial alignment / image registration
be able to apply dynamic time-warping for temporal aligment of time-series
be able to use k-means multi-modal image segmentation.
be able to normalize data in different ways

Relevant Lecture¶

Lecture 4

Relevant Sections in course book¶

Chapters 5, 6, 7, and 8

Additional Material¶

Visualization and plotting with matplotlib
Image processing with scikit-image
Dynamic time warping with simpledtw
scikit-learn page on k-means

Tutorial¶

Loading and processing images¶

Tutorial. Image processing with scikit-image.

The code below reads image from the file, rotates it, transforms to different color space, and finally plot the result images. Run the code and get yourself familiar with libraries and functions (skimage.transform and skimage.color).

import numpy as np
import matplotlib.image as image
import matplotlib.pyplot as plt
from skimage.transform import rotate
from skimage.color import rgb2hsv

# loading image
img = image.imread('images/bear1.jpg')
# rotating image 90 degrees
img_rotated = rotate(img, 90)
# transform color space from RGB to HSV
img_hsv = rgb2hsv(img)

# display resulting images
f = plt.figure(figsize=(10,5))
f.add_subplot(1,3,1)
plt.imshow(img)
f.add_subplot(1,3,2)
plt.imshow(img_rotated)
f.add_subplot(1,3,3)
plt.imshow(img_hsv)

<matplotlib.image.AxesImage at 0x7f9b3cf40fd0>

Plotting historams¶

Tutorial. Calculating and visualizing histograms.

The code below reads image from the file, transforms it to gray scale, calculates histograms. Run the code and get yourself familiar with 1D and 2D histograms (matplotlib.pyplot.hist and numpy.histogram2d). Study what does these 1D and 2D histograms means in image context.

from skimage.color import rgb2gray
# loading image
img = image.imread('images/bear2.jpg')
# transform to gray scale
img_gray = rgb2gray(img)

# display images
f1 = plt.figure(figsize=(10,10))
f1.add_subplot(1,2,1)
h1 = plt.imshow(img)
f1.add_subplot(1,2,2)
h1 = plt.imshow(img_gray, cmap=plt.get_cmap('gray'))
plt.show()

# calculate and plot images and 1D/2D histogram using 20 bins
nbins = 20
f2 = plt.figure(figsize=(10,5))
f2.add_subplot(1,2,1)
hist1d = plt.hist(img_gray.flatten(), bins=nbins)
f2.add_subplot(1,2,2)
hist2d = np.histogram2d(img_gray.flatten(), img_gray.flatten(), bins=nbins)[0]
nz = hist2d > 0
hist2d[nz] = np.log(hist2d[nz])
plt.imshow(hist2d, interpolation='nearest')

<matplotlib.image.AxesImage at 0x7f9b3cd087c0>

Dynamic time warping¶

Tutorial. Dynamic time warping between time-series.

The code below generates two univariate time-series and then match them using Dynamic time warping (DTW) method. Run the code and get yourself familiar with DTW simpledtw. Study how DTW works and what are the inputs and outputs in dtw()-function. Outputs are also visualized in next code snippet.

DTW: toy example¶

import simpledtw

ts1 = np.array([4, 3.5, 3.7, 4, 4.4, 5, 6, 6.5]) # 1st time-series
ti1 = np.array(range(0, len(ts1))) # time indices
ts2 = np.array([2, 2.7, 2.5, 4.5, 5, 5.7]) # 2nd time-series
ti2 = np.array(range(2, len(ts1))) # time indices
matches, cost, mapping1, mapping2, matrix = simpledtw.dtw(np.array([ti1, ts1]).T, np.array([ti2,ts2]).T) # calcute DTW distance between time-series
print('Distance (warping cost):', cost)
print('Dynamic programming matrix:\n', matrix)
print('Mapping 1', mapping1)
print('Mapping 2', mapping2)

Distance (warping cost): 11.153179524695364
Dynamic programming matrix:
 [[ 2.82842712  6.09798367 10.36998554 15.39492335 21.47768588 28.68115727]
 [ 4.63120276  4.98249305  8.14477071 12.26787633 17.48802959 23.87864771]
 [ 6.33120276  6.04541632  7.31487381 10.41970874 14.62565691 20.01082172]
 [ 8.56727074  7.34541632  7.84819196  9.37642662 12.53870428 16.88496704]
 [11.69137061  9.31772462  9.24541632  8.85317952 10.94124083 14.21079737]
 [15.9340113  12.36567475 11.93799873  9.35317952  9.85317952 11.97214153]
 [21.59086555 16.82549537 15.9691276  11.15595516 10.35317952 10.89721018]
 [28.31767757 22.34274106 20.9691276  13.98438229 12.15595516 11.15317952]]
Mapping 1 [[0], [0], [1], [2], [3], [3], [4], [5]]
Mapping 2 [[0, 1], [2], [3], [4, 5], [6], [7]]

DTW: plotting the results¶

f = plt.figure(figsize=(10,5))
f.add_subplot(1,2,1)

# plot data
plt.plot(ti1, ts1, '-o')
plt.plot(ti2, ts2, '-o')

# plot connections between points
for i, indices in enumerate(mapping1):
    for j in indices:
        plt.plot([ti1[i],ti2[j]],[ts1[i],ts2[j]], 'k')

for i, indices in enumerate(mapping2):
    for j in indices:
        plt.plot([ti1[j],ti2[i]],[ts1[j],ts2[i]], 'k')  

# plot dp matrix
ax = f.add_subplot(1,2,2)
ax.imshow(matrix) 
for i in range(matrix.shape[0]):
    for j in range(matrix.shape[1]):
        text = ax.text(j, i, np.round(matrix[i, j], 2),
                       ha="center", va="center", color="w")

plt.show()

K-means clustering¶

Tutorial. Clustering with k-means algorithm.

The code below defines k-means clustering algorithm and applied it to syntetic 2D data. Run the code and study how k-means works (sklearn.cluster.Kmeans), what are the inputs and outputs of kmeans_clustering()-function, how can you interpret the result of toy example.

from sklearn.cluster import KMeans

def kmeans_clustering(X, k=3, maximum_iterations=200):
    n, d = X.shape    # number of examples n and number of features d
    cluster_centers = np.zeros((k,d))    # Store the resulting clustering centers in the rows of this k x d matrix
    cluster_labels = np.zeros(n)    # Store here the resulting n cluster indices / labels (assigned for each example)

    k_means = KMeans(n_clusters = k, max_iter = maximum_iterations).fit(X)  # Fit k-means with k clusters and using max_iter maximum iterations
    cluster_centers = k_means.cluster_centers_  # cluster centers
    cluster_indices = k_means.labels_  # cluster label indices for each example
    return cluster_centers, cluster_indices

K-means: toy example¶

from sklearn.datasets import make_blobs

# generate 2D clusters with 5 centers
X, y = make_blobs(n_samples=200, centers=5, n_features=2, random_state=1)

# fit k-means fit 5 clusters
cluster_means, cluster_indices = kmeans_clustering(X, k = 5)

# visualize data and clustering results
plt.scatter(X[:, 0], X[:, 1], marker='o', c=cluster_indices,
            s=25, edgecolor='k')
plt.scatter(cluster_means[:,0], cluster_means[:,1], marker='x', s=100, color = 'r')

<matplotlib.collections.PathCollection at 0x7f9b35f16370>

-----------------------------------------------------------------------¶

Assignments¶

1. Spatial Alignment: mutual Information for image registration¶

Introduction¶

In this assigment, we will take a look at mutual information between two images. For background, look at first part of the Lecture 4 (spatial alignment). Also, Chapter 5 in course book is relevant.

The image data¶

The image data here are from two different magnetic resonance imaging sources ('T1-weighted-MRI.jpg', and 'T2-weighted-MRI.jpg'). Task is to transform the images and calculate mutual information based on the joint histogram.

Assignment. Problem 1.1.

1) Complete the code below to rotate second MRI image 't2_img' by 45 degrees.

2) Visualize original images and the resulting rotated image.

Hint: Your output image should look like in the cell below

# Example image required for problem 1.1
from IPython.display import Image
Image(filename='images/task1.1.png')

import numpy as np
import matplotlib.image as img
import matplotlib.pyplot as plt
from skimage.transform import rotate

# Answer to problem 1.1 in this cell

# load MRI images ('T1-weighted-MRI.jpg' and T2-weighted-MRI.jpg) as pixel array
# YOUR CODE HERE
t1_img = img.imread('images/T1-weighted-MRI.jpg')
t2_img = img.imread('images/T2-weighted-MRI.jpg')

# rotate image 't2_img' by 45 degrees
t2_img_45 = rotate(t2_img, 45)

# display all 3 array of pixels as an image
## Your code here
f = plt.figure(figsize=(10,5))
f.add_subplot(1,3,1)
plt.imshow(t1_img, cmap=plt.get_cmap('gray'))
f.add_subplot(1,3,2)
plt.imshow(t2_img, cmap=plt.get_cmap('gray'))
f.add_subplot(1,3,3)
plt.imshow(t2_img_45, cmap=plt.get_cmap('gray'))

<matplotlib.image.AxesImage at 0x7f9b34406d90>

# Your points for Problem 1.1 will be given in this cell. Do not modify it.

1D Histograms¶

Assignment. Problem 1.2.

1) Calculate and plot 1D histograms of three images above using 20 histogram bins.

2) Compare the results. What can you say about histograms? Why first two are different and last two almost similar?

Hint: Your output image should look like in the cell below

# Example image required for problem 1.2
from IPython.display import Image
Image(filename='images/task1.2.png')

# Answer to Problem 1.2.1 in this cell 
# YOUR CODE HERE
nbins = 20
f = plt.figure(figsize=(12,5))
f.add_subplot(1,3,1)
h1 = plt.hist(t1_img.flatten(), bins=nbins) # 1D histogram of t1_img
f.add_subplot(1,3,2)
h2 = plt.hist(t2_img.flatten(), bins=nbins) # 1D histogram of t2_img
f.add_subplot(1,3,3)
h3 = plt.hist(t2_img_45.flatten(), bins=nbins) # 1D histogram of t2_img_45

# Your points for Problem 1.2.1 will be given in this cell. Do not modify it.

1.2.2: What can you say about histograms? Why first two are different and last two similar?¶

Answer to 1.2.2 in the cell below

The original images produce different histograms, due to the difference in their pixel values.

The last two histograms however are very similar because rotation is a transformation that conserves those pixel values and only changes their location.

2D Histograms¶

Assignment. Problem 1.3.

1) Calculate and plot 2D histograms of images below using 20 histogram bins (interpolation='nearest'). Hint: Your output image should look like in the cell below

2) Compare the results. What can you say about 2D histograms? Why first one is diagonal and last two different compared to 1.2?

# Example image required for problem 1.3
from IPython.display import Image
Image(filename='images/task1.3.png')

# Answer to Problem 1.3.1. in this cell

nbins = 20

# Plot three histogram between
    # 't1_img' and 't1_img'
    # 't1_img' and 't2_img'
    # 't1_img' and 't2_img_45'
    
    # Do realize, that if you use a different library to create the histograms, the image may look different.
    # Check that your image has the same type of data as the one in output hints. 
    # Your answer may be correct even if the images do not look the same. 
    # The output hints have been made using numpy. 

# YOUR CODE HERE

H1 = np.histogram2d(t1_img.flatten(), t1_img.flatten(), bins=nbins)[0] # 2D histogram t1_img, t1_img
H2 = np.histogram2d(t1_img.flatten(), t2_img.flatten(), bins=nbins)[0] # 2D histogram t1_img, t2_img
H3 = np.histogram2d(t1_img.flatten(), t2_img_45.flatten(), bins=nbins)[0] # 2D histogram t1_img, t2_img_45

hist1 = np.copy(H1)
hist1[H1 > 0] = np.log(H1[H1 > 0]) # Create a copy of H1 and then take logarithm of non-zero elements
hist2 = np.copy(H2)
hist2[H2 > 0] = np.log(H2[H2 > 0]) # Create a copy of H2 and then take logarithm of non-zero elements
hist3 = np.copy(H3)
hist3[H3 > 0] = np.log(H3[H3 > 0]) # Create a copy of H3 and then take logarithm of non-zero elements

# Hint! See tutorial for help

f = plt.figure(figsize=(12,5))
f.add_subplot(1,3,1)
plt.imshow(hist1, interpolation='nearest')
f.add_subplot(1,3,2)
plt.imshow(hist2, interpolation='nearest')
f.add_subplot(1,3,3)
plt.imshow(hist3, interpolation='nearest')

<matplotlib.image.AxesImage at 0x7f9b27f4b280>

# Your points for Problem 1.3.1 will be given in this cell. Do not modify it.

1.3.2: Compare the results. What can you say about 2D histograms? Why first one is diagonal and last two different compared to 1.2?¶

Answer to 1.3.2 in the cell below

In the first 2D histogram, the 2 images were the same resulting in it being diagonal i.e. the intersecion between the bins are similar.

The other histograms start showing some noticeable differences between the reference and target images.

Mutual information of histograms¶

Assignment. Problem 1.4.

Complete the code below to calculate and return mutual information of joint 2D histogram. Mutual information is defined as: \begin{equation*} MI(X;Y) = \sum_{y \in Y} \sum_{x \in X} p(x,y) log\bigg(\frac{p(x,y)}{p(x)p(y)}\bigg) \end{equation*} Also relevant material can be found from Lecture 4 (pages 5-9)

def mutual_information(hgram):
    """ Mutual information for joint histogram
    """
    # YOUR CODE HERE
    # Convert bins counts to probability values
    pxy = hgram / np.sum(hgram)
    # calculate marginal for x over y
    px = np.sum(pxy, axis=0)
    # calculate marginal for y over x
    py = np.sum(pxy, axis=1)
    # Broadcast to multiply marginals
    px_py = px[:, None] * py[None, :]
    ## Now we can do the calculation using the probability values (pxy) and mutiplied marginals (px_py) 2D arrays
    # hint: Only non-zero pxy values contribute to the sum
    nz = pxy > 0
    mi = np.sum(pxy[nz] * np.log(pxy[nz] / px_py[nz]))
    return mi

# Your points for Problem 1.4 will be given in this cell. Do not modify it.

Rotated images and mutual information¶

Assignment. Problem 1.5.

Complete the code below to calculate mutual information for rotated images using your solution.

1) Generate 19 more rotated images of 't2_img' in the range of -45 to +45 degrees with increament of 5 degrees.

2) Compute histograms between first mri image ('t1_img') and rotated images. Use bins = nbins

3) Calculate mutual information for each of these histograms (using function implemented above).

4) Plot the mutual infomation values as function of rotation angles. Hint: Your output image should look like in the cell below

5) In which angle mutual information is maximized? Think by yourself why is that.

# Example image required for problem 1.5.4
from IPython.display import Image
Image(filename='images/task1.5.png')

# Problem 1.5.1.
# generate 19 more rotated images of 't2_img' in the range of -45 to +45 degrees with increament of 5 degrees
rotated_images = [] # == save rotated images
labels = [] # == save angles of rotation for corresponding image
# YOUR CODE HERE
labels = [x for x in range(-45, 46, 5)]
rotated_images = [rotate(t2_img, x) for x in labels]

# Problem 1.5.2.
# compute histograms between t1_img and all rotated t2_img ('rotated_images')
histograms = [] # == save histograms
## Your code here
histograms = [np.histogram2d(t1_img.flatten(), r_img.flatten(), bins=nbins)[0] for r_img in rotated_images]

# Problem 1.5.3.
# calculate mutual information of each histogram
mi = [mutual_information(hgram) for hgram in histograms]
## Your code here

# Problem 1.5.4.
# plot mutual information of rotated images
## Your code here
plt.plot(labels, mi)

[<matplotlib.lines.Line2D at 0x7f9b266a3460>]

# Your points for Problems 1.5.1 and 1.5.2 will be given in this cell. Do not modify it.

# Your points for Problems 1.5.3 and 1.5.4 will be given in this cell. Do not modify it.

# Problem 1.5.5.
# Give answer as a integer corresponding to the angle. 

answer = labels[np.argmax(mi)]

# Your points for Problem 1.5.5 will be given in this cell. Do not modify it.

2. Temporal Alignment: Dynamic Time Warping¶

Introduction¶

In this assigment, simpledtw library is used to calculate the dynamic time warping. This lirary can be directly imported from the exercise folder. Get familiar with the library (if not done yet) and complete the tasks below. For the background, take a look at second part of Lecture 4 (temporal aligment). Also Chapter 6. in course book is relevant.

Hand gesture times series data¶

In this assignment, example of hand gestures recorded with wearable IMU devices are used to study DTW. This dataset includes multi-dimensional measurements but mean of 3-exis acceleration sensor is used as a time-series.

Assignment. Problem 2.1.

Complete the code below to calculate DTW-based similarity between reference (reference_gesture) and unknown (gesture1, gesture2, gesture3) gestures.

1) Plot all four gestures. Hint: Your output image should look like in the cell below

2) Calculate DTW similarity between reference and unknown gestures (use code from tutorial above).

3) Print warping cost between reference and unknown gestures.

4) Which one of the unkown gestures belong to same class as the reference gesture? Why?

# Example image required for problem 2.1.1
from IPython.display import Image
Image(filename='images/task2.1.png')

import simpledtw
import numpy as np

# load data
g1_1 = np.genfromtxt('data/gesture1.txt', delimiter=' ')
g1_2 = np.genfromtxt('data/gesture2.txt', delimiter=' ')
g2_1 = np.genfromtxt('data/gesture3.txt', delimiter=' ')
g2_2 = np.genfromtxt('data/gesture4.txt', delimiter=' ')

# pickup time-stamps and mean feature
reference_gesture = np.array([(g1_1[:,0]-g1_1[0,0])/1000.0, np.mean(g1_1[:, 3:6], axis=1)])
gesture1 = np.array([(g1_2[:,0]-g1_2[0,0])/1000.0, np.mean(g1_2[:, 3:6], axis=1)])
gesture2 = np.array([(g2_1[:,0]-g2_1[0,0])/1000.0, np.mean(g2_1[:, 3:6], axis=1)])
gesture3 = np.array([(g2_2[:,0]-g2_2[0,0])/1000.0, np.mean(g2_2[:, 3:6], axis=1)])

# Problem 2.1.1.
# Plot all four gestures
# YOUR CODE HERE
f = plt.figure(figsize=(16,5))
f.add_subplot(1,4,1)
plt.plot(reference_gesture[0], reference_gesture[1])
f.add_subplot(1,4,2)
plt.plot(gesture1[0], gesture1[1])
f.add_subplot(1,4,3)
plt.plot(gesture2[0], gesture2[1])
f.add_subplot(1,4,4)
plt.plot(gesture3[0], gesture3[1])
plt.show()

# Problem 2.1.2.
# calculate DTW similarity between reference and unknown gestures
## Your code here
matches1, cost1, mapping_1_1, mapping_1_2, matrix1 = simpledtw.dtw(reference_gesture.T, gesture1.T) #Reference and gesture 1
matches2, cost2, mapping_2_1, mapping_2_2, matrix2 = simpledtw.dtw(reference_gesture.T, gesture2.T) #Reference and gesture 2
matches3, cost3, mapping_3_1, mapping_3_2, matrix3 = simpledtw.dtw(reference_gesture.T, gesture3.T) #Reference and gesture 3

# Problem 2.1.3.
# print warping cost between reference and unknown gestures
## Your code here

print('warping cost between references and...')
for i, cost in enumerate([cost1, cost2, cost3]):
    print(f'\tgesture {i+1}: {cost}')
        
        
# f = plt.figure(figsize=(30,10))
# for m_idx, matrix in enumerate([matrix1, matrix2, matrix3]):
#     ax = f.add_subplot(1,3,m_idx+1)
#     ax.imshow(matrix) 
#     for i in range(matrix.shape[0]):
#         for j in range(matrix.shape[1]):
#             text = ax.text(j, i, np.round(matrix[i, j], 1), ha="center", va="center", color="w")
# plt.show()

warping cost between references and...
	gesture 1: 6.428572992508663
	gesture 2: 25.78906643723184
	gesture 3: 51.367616886673986

# Your points for Problems 2.1.1. - 2.1.3. will be given in this cell. Do not modify it.

# Problem 2.1.4.
# a = gesture1, b = gesture2, c = gesture3
answer = "a"

# Your points for Problem 2.1.4. will be given in this cell. Do not modify it.

Assignment. Problem 2.2.

1) Plot mappings (ie., warpings) between reference (reference_gesture) and unknown (gesture1, gesture2, gesture3) gestures. Hint: Your output image should look like in the cell below

2) Compare the results. What can be said about the mappings?

# Example image required for problem 2.2.1
import matplotlib.pyplot as plt
fig, axs = plt.subplots(1,3)
fig.set_size_inches(16, 8)
axs[0].imshow(plt.imread('images/task2.2.1.png'))
axs[1].imshow(plt.imread('images/task2.2.2.png'))
axs[2].imshow(plt.imread('images/task2.2.3.png'))
for ax in axs:
    ax.axes.get_xaxis().set_visible(False)
    ax.axes.get_yaxis().set_visible(False)
plt.show()

# Problem 2.2.1: Plot mapping
def mapplot(gesture1, gesture2, mapping1, mapping2):
    # Implement function which makes the required plots
    # YOUR CODE HERE
    plt.plot(gesture1[:, 0], gesture1[:, 1], '-o')
    plt.plot(gesture2[:, 0], gesture2[:, 1], '-o')

    for i, indices in enumerate(mapping1):
        for j in indices:
            plt.plot([gesture1[i, 0],gesture2[j, 0]],[gesture1[i, 1],gesture2[j, 1]], 'k')

    for i, indices in enumerate(mapping2):
        for j in indices:
            plt.plot([gesture1[j, 0],gesture2[i, 0]],[gesture1[j, 1],gesture2[i, 1]], 'k')  
            
    plt.show()
    
    
mapplot(reference_gesture.T, gesture1.T, mapping_1_1, mapping_1_2)
mapplot(reference_gesture.T, gesture2.T, mapping_2_1, mapping_2_2)
mapplot(reference_gesture.T, gesture3.T, mapping_3_1, mapping_3_2)

# Your points for Problem 2.2.1. will be given in this cell. Do not modify it.

Student plots should look like this:

Problem 2.2.2: Compare the results. What can be said about the mappings?¶

Answer in the cell below

The mapping confirms that gesture1 belongs to the same class as the reference gesture because of the much shorter distances compared to the other gestures.

3. Semantic Alignment: Clustering¶

Introduction¶

In this assignment, k-means clustering ensembles are used to segment color image. Different color spaces are formed and clustered independently in pixel level. The results are then fused using another k-means applied to joint feature presentation formed from the identity vectors of individual algorithms. For the background, take a look at third part of Lecture 4 (semantic alignment). Also, Chapter 7. in the course book is relevant.

Image transforms to different color spaces¶

Assignment. Problem 3.1.

1) Complete the code below to transform input image to different color spaces of RGB, HSV, LAB, YUV, XYZ, LUV using the methods in scikit-image library

2) Min-Max normalize transformed images to [0, 1]

3) Plot all six images (original and normalized transformed images). Remember that you are no longer plotting RGB images! Hint: Your output image should look like in the cell below

# Example image required for problem 3.1.3
from IPython.display import Image
Image(filename='images/task3.1.png')

import numpy as np
import matplotlib.image as img
import matplotlib.pyplot as plt
# Import relevant functions from skimage
# YOUR CODE HERE
from skimage.color import rgb2hsv, rgb2lab, rgb2yuv, rgb2xyz, rgb2luv 

# load image 'landscape.jpg' as pixel array
f1_img = img.imread('images/landscape.jpg')

# Problem 3.1.1.
# 1) convert image 'f1_img' int different color maps from rgb
f2_img = rgb2hsv(f1_img) # into HSV
f3_img = rgb2lab(f1_img) # into LAB
f4_img = rgb2yuv(f1_img) # into YUV
f5_img = rgb2xyz(f1_img) # into XYZ
f6_img = rgb2luv(f1_img) # into LUV

# Problem 3.1.2.
# 2) Min-Max normalize f2_img, ..., f6_img to [0, 1]
f2_img = (f2_img - np.min(f2_img)) / (np.max(f2_img) - np.min(f2_img))
f3_img = (f3_img - np.min(f3_img)) / (np.max(f3_img) - np.min(f3_img))
f4_img = (f4_img - np.min(f4_img)) / (np.max(f4_img) - np.min(f4_img))
f5_img = (f5_img - np.min(f5_img)) / (np.max(f5_img) - np.min(f5_img))
f6_img = (f6_img - np.min(f6_img)) / (np.max(f6_img) - np.min(f6_img))

# Problem 3.1.3.
# display all arrays of pixels as images (original and normalized transformed images)
## Your code here
imgs = [f1_img, f2_img, f3_img, f4_img, f5_img, f6_img]
f = plt.figure(figsize=(15,7))
for i, img in enumerate(imgs):
    ax = f.add_subplot(2,3,i + 1)
    ax.imshow(img)

# Your points for Problem 3.1 will be given in this cell. Do not modify it.

Image segmentation by different color spaces¶

Assignment. Problem 3.2.

Complete the code below to apply k-means clustering separately to different color spaces (for all images calculated above i.e., original and all normalized transformed images). Hint: use function "kmeans_clustering" represented in tutorial section above. Use 4 cluster and pixel level features (e.g., in RGB each pixel is presented as 3-dimensional vector, and there should be 273280 vectors). Plot the segmented images.

Hint: Your output image should look something like in the cell below

# Example image required for problem 3.2
from IPython.display import Image
Image(filename='images/task3.2.png')

np.random.seed(0) # Do not change or remove

# Make feature vectors from the pixels 
# YOUR CODE HERE
vecs = []
for img in imgs:
    vecs.append(img.reshape((-1, 3)))
    
k = 4 # number of clusters to form

# Perform k-means clustering on all six images seperately and compute resulting images

# Do the clustering for first image here
## Your code here
cluster_means1, cluster_indices1 = kmeans_clustering(vecs[0], k)
res_img1 = cluster_indices1.reshape(imgs[0].shape[:2])

# Your points for first clustering will be given in this cell. Do not modify it.

np.random.seed(0) # Do not change or remove

# Do the clustering for second image here
# YOUR CODE HERE
cluster_means2, cluster_indices2 = kmeans_clustering(vecs[1], k)
res_img2 = cluster_indices2.reshape(imgs[1].shape[:2])

# Your points for second clustering will be given in this cell. Do not modify it.

np.random.seed(0) # Do not change or remove

# Do the clustering for third image here
# YOUR CODE HERE
cluster_means3, cluster_indices3 = kmeans_clustering(vecs[2], k)
res_img3 = cluster_indices3.reshape(imgs[2].shape[:2])

# Your points for third clustering will be given in this cell. Do not modify it.

np.random.seed(0) # Do not change or remove

# Do the clustering for fourth image here
# YOUR CODE HERE
cluster_means4, cluster_indices4 = kmeans_clustering(vecs[3], k)
res_img4 = cluster_indices4.reshape(imgs[3].shape[:2])

# Your points for fourth clustering will be given in this cell. Do not modify it.

np.random.seed(0) # Do not change or remove

# Do the clustering for fith image here
# YOUR CODE HERE
cluster_means5, cluster_indices5 = kmeans_clustering(vecs[4], k)
res_img5 = cluster_indices5.reshape(imgs[4].shape[:2])

# Your points for fifth clustering will be given in this cell. Do not modify it.

np.random.seed(0) # Do not change or remove

# Do the clustering for sixth image here
# YOUR CODE HERE
cluster_means6, cluster_indices6 = kmeans_clustering(vecs[5], k)
res_img6 = cluster_indices6.reshape(imgs[5].shape[:2])

# Your points for sixth clustering will be given in this cell. Do not modify it.

# Plot all k-means clustering resulting images
# YOUR CODE HERE
res_imgs = [res_img1, res_img2, res_img3, res_img4, res_img5, res_img6]
f = plt.figure(figsize=(15,7))
for i, img in enumerate(res_imgs):
    ax = f.add_subplot(2,3,i + 1)
    ax.imshow(img)

# Your points for the plot will be given in this cell. Do not modify it.

Image segmentation by fusion of color spaces¶

Assignment. Problem 3.3.

Complete the code below to apply k-means clustering to fuse the results of different color spaces from previous tasks. Concatenate the clustering results from each k-means model to joint feature vector. Normalize features between 0 and 1. Plot the original image and fused segmented image.

Hint: Your output image should look something like in the cell below

# Example image required for problem 3.3
from IPython.display import Image
Image(filename='images/task3.3.png')

# concatenate cluster features to joint feature presentation (Output size should be (273280, 6) with 6 dimensions corresponding to 6 colour spaces)
# YOUR CODE HERE
X = np.concatenate([img.reshape((-1, 1)) for img in res_imgs], axis=1) # joint feature presentation

# normalize 0 to 1 (0-3 clusters)
## Your code here
X = (X - np.min(X)) / (np.max(X) - np.min(X))

# Perform k-means clustering on fused image and compute resulting image
np.random.seed(0) # Do not change or remove
## Your code here
cluster_means, cluster_indices = kmeans_clustering(X, k)
res_img = cluster_indices.reshape(imgs[0].shape[:2])

# Plot original and k-means clustering resulting image
## Your code here
f = plt.figure(figsize=(15,7))
ax = f.add_subplot(1,2,1)
plt.imshow(imgs[0])
ax = f.add_subplot(1,2,2)
plt.imshow(res_img)

<matplotlib.image.AxesImage at 0x7f9b265f46a0>

# Your points for Problem 3.3 joint feature implementation will be given in this cell. Do not modify it.

# Your points for Problem 3.3 clustering and plot implementation will be given in this cell. Do not modify it.

4. Radiometric Normalization¶

Introduction¶

In this assigment, different radiometric scaling approaches are studied using heart disease dataset. For the background, take a look at fourth part of Lecture 4 (radiometric normalization). Also, Chapter 8. in course book is relevant.

Dataset¶

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

data = pd.read_csv('data/heart-disease.csv')
data.head(10)

Scales of measurements¶

Assignment. Problem 4.1.

Run the code below and answer the questions below.

1) What are the different scales of measurement types?

2) What are the scales of variables below?

Variables are introduced in https://archive.ics.uci.edu/ml/datasets/heart+disease

data[['age','sex','trestbps','chol']].head(10)

# Problem 4.1.1.
scales = ['interval', 'nominal', 'ratio', 'ratio'] # Give the names of the scales as a list. Give each scale name as a string

# Problem 4.1.2.
# Give scale name as a string 
age = scales[0] # scale name 
sex = scales[1] # scale name
trestbps = scales[2] # scale name
chol = scales[3] # scale name

# Your points for Problem 4.1.1 will be given in this cell. Do not modify it.

# Your points for Problem 4.1.2 age scale will be given in this cell. Do not modify it.

# Your points for Problem 4.1.2 sex scale will be given in this cell. Do not modify it.

# Your points for Problem 4.1.2 trestbps scale will be given in this cell. Do not modify it.

# Your points for Problem 4.1.2 chol scale will be given in this cell. Do not modify it.

Degree-of-similarity scales¶

# Lets use only subset of data in rest of the problems
subdata = data[['age','trestbps', 'chol']].head(5)
subdata

Euclidean distance¶

Assignment. Problem 4.2.

Complete the code below to calculate euclidean distance based similarity matrix for 'subdata'.

Euclidean distance for vectors a and b with dimension M is defined as:

\begin{equation*} d(a, b)=\sqrt{\sum_{m=1}^M(a_m - b_m)^2} \end{equation*}

def euclidean_distance(a, b):
    # YOUR CODE HERE
    distance_matrix = np.sqrt(np.sum((a-b)**2))
    return distance_matrix

d = subdata.to_numpy()
distance_matrix = np.zeros((5, 5))
for i in range(distance_matrix.shape[0]):
    for j in range(distance_matrix.shape[1]):
        distance_matrix[i,j] = euclidean_distance(d[i,:], d[j,:])
print('Euclidean distances: \n', distance_matrix)

Euclidean distances: 
 [[ 0.         55.22680509 25.63201124 34.49637662 39.37003937]
 [55.22680509  0.         69.63476143 55.64171097 91.10433579]
 [25.63201124 69.63476143  0.         37.96050579 37.42993454]
 [34.49637662 55.64171097 37.96050579  0.         46.17358552]
 [39.37003937 91.10433579 37.42993454 46.17358552  0.        ]]

# Your points for Problem 4.2 will be given in this cell. Do not modify it.

City-block distance distance¶

Assignment. Problem 4.3.

Complete the code below to calculate city-block distance based similarity matrix for 'subdata'.

City-block distance for vectors a and b with dimension M is defined as:

\begin{equation*} d(a, b)=\sum_{m=1}^M |a_m - b_m| \end{equation*}

def city_block_distance(a, b):
    # YOUR CODE HERE
    distance_matrix = np.sum(abs(a-b))
    return distance_matrix

d = subdata.to_numpy()
distance_matrix = np.zeros((5, 5))
for i in range(distance_matrix.shape[0]):
    for j in range(distance_matrix.shape[1]):
        distance_matrix[i,j] = city_block_distance(d[i,:], d[j,:])
print('City-block distances: \n', distance_matrix)

City-block distances: 
 [[  0.  72.  33.  58.  66.]
 [ 72.   0.  97.  96. 138.]
 [ 33.  97.   0.  61.  61.]
 [ 58.  96.  61.   0.  50.]
 [ 66. 138.  61.  50.   0.]]

# Your points for Problem 4.3 will be given in this cell. Do not modify it.

Chebyshev distance¶

Assignment. Problem 4.4.

Complete the code below to calculate Chebyshev distance based similarity matrix.

Chebyshev distance for vectors a and b with dimension M is defined as:

\begin{equation*} d(a, b)= max_m (|a_m - b_m|) \end{equation*}

def chebyshev_distance(a, b):
    # YOUR CODE HERE
    distance_matrix = np.max(abs(a-b))
    return distance_matrix

d = subdata.to_numpy()
distance_matrix = np.zeros((5, 5))
for i in range(distance_matrix.shape[0]):
    for j in range(distance_matrix.shape[1]):
        distance_matrix[i,j] = chebyshev_distance(d[i,:], d[j,:])       
print('Chebyshev distances: \n', distance_matrix)

Chebyshev distances: 
 [[ 0. 53. 25. 26. 29.]
 [53.  0. 57. 36. 82.]
 [25. 57.  0. 30. 26.]
 [26. 36. 30.  0. 46.]
 [29. 82. 26. 46.  0.]]

# Your points for Problem 4.4 will be given in this cell. Do not modify it.

Parametric normalization¶

Z-transform scaling¶

Assignment. Problem 4.5.

Complete the code below to calculate Z-transfrom scaling from given data, i.e., zero mean unit standard deviation scaling.

Z-transform:

\begin{equation*} y = \frac{x - \mu}{\sigma}, \end{equation*} where $\mu$ is the sample mean, and $\sigma$ is the standard deviation.

subdata = data[['age','trestbps', 'chol']]

def z_transform(data):
    # YOUR CODE HERE
    scaled_data = (data - data.mean())/data.std()
    return scaled_data # Function should return pandas DataFrame
    
scaled_data = z_transform(subdata) 
answer = scaled_data.head(10)

# Your points for Problem 4.5 will be given in this cell. Do not modify it.

Min-Max scaling¶

Assignment. Problem 4.6.

Complete the code below to calculate min-max scaling for given data.

min-max:

\begin{equation*} y = \frac{x -\min (x)}{\max (x)-\min (x)} \end{equation*}

subdata = data[['age','trestbps', 'chol']]

def min_max(data):
    # YOUR CODE HERE
    scaled_data = (data - data.min()) / (data.max() - data.min())
    return scaled_data # Function should return pandas DataFrame


scaled_data = min_max(subdata)
answer = scaled_data.head(10)

# Your points for Problem 4.6 will be given in this cell. Do not modify it.

	age	sex	cp	trestbps	chol	fbs	restecg	thalach	exang	oldpeak	slope	ca	thal	num
0	63.0	1.0	1.0	145.0	233.0	1.0	2.0	150.0	0.0	2.3	3.0	0.0	6.0	0
1	67.0	1.0	4.0	160.0	286.0	0.0	2.0	108.0	1.0	1.5	2.0	3.0	3.0	2
2	67.0	1.0	4.0	120.0	229.0	0.0	2.0	129.0	1.0	2.6	2.0	2.0	7.0	1
3	37.0	1.0	3.0	130.0	250.0	0.0	0.0	187.0	0.0	3.5	3.0	0.0	3.0	0
4	41.0	0.0	2.0	130.0	204.0	0.0	2.0	172.0	0.0	1.4	1.0	0.0	3.0	0
5	56.0	1.0	2.0	120.0	236.0	0.0	0.0	178.0	0.0	0.8	1.0	0.0	3.0	0
6	62.0	0.0	4.0	140.0	268.0	0.0	2.0	160.0	0.0	3.6	3.0	2.0	3.0	3
7	57.0	0.0	4.0	120.0	354.0	0.0	0.0	163.0	1.0	0.6	1.0	0.0	3.0	0
8	63.0	1.0	4.0	130.0	254.0	0.0	2.0	147.0	0.0	1.4	2.0	1.0	7.0	2
9	53.0	1.0	4.0	140.0	203.0	1.0	2.0	155.0	1.0	3.1	3.0	0.0	7.0	1

	age	trestbps	chol
0	63.0	145.0	233.0
1	67.0	160.0	286.0
2	67.0	120.0	229.0
3	37.0	130.0	250.0
4	41.0	130.0	204.0

exercise3 (Score: 9.75 / 10.0)

Multimodal Data Fusion - Exercise 3 - Data Alignment¶

Learning goals¶

Relevant Lecture¶

Relevant Sections in course book¶

Additional Material¶

Tutorial¶

Loading and processing images¶

Plotting historams¶

Dynamic time warping¶

DTW: toy example¶

DTW: plotting the results¶

K-means clustering¶

K-means: toy example¶

-----------------------------------------------------------------------¶

Assignments¶

1. Spatial Alignment: mutual Information for image registration¶

Introduction¶

The image data¶

1D Histograms¶

1.2.2: What can you say about histograms? Why first two are different and last two similar?¶

2D Histograms¶

1.3.2: Compare the results. What can you say about 2D histograms? Why first one is diagonal and last two different compared to 1.2?¶

Mutual information of histograms¶

Rotated images and mutual information¶

2. Temporal Alignment: Dynamic Time Warping¶

Introduction¶

Hand gesture times series data¶

Problem 2.2.2: Compare the results. What can be said about the mappings?¶

3. Semantic Alignment: Clustering¶

Introduction¶

Image transforms to different color spaces¶

Image segmentation by different color spaces¶

Image segmentation by fusion of color spaces¶

4. Radiometric Normalization¶

Introduction¶

Dataset¶

Scales of measurements¶

Degree-of-similarity scales¶

Euclidean distance¶

City-block distance distance¶

Chebyshev distance¶

Parametric normalization¶

Z-transform scaling¶

Min-Max scaling¶